Local illumination and color compensation without explicit signaling

ABSTRACT

There are provided method and apparatus for local illumination and color compensation without explicit signaling. An apparatus includes an encoder for encoding a picture by enabling at least one of color compensation and illumination compensation of at least one color component of a prediction for an image block of the picture without using explicit signaling. The method comprises at least the step of enabling the encoding of a picture described above in the apparatus, again without use of explicit signaling. In the description of an specific embodiment, this enabling of the color compensation and the illumination compensation is based on at least one of a group of pixels and a local signal block corresponding to the picture. Similar descriptions are provided for a method and apparatus for decoding the signal the encoded signal. There are provided method and apparatus for local illumination and color compensation without explicit signaling. An apparatus includes an encoder for encoding a picture by enabling at least one of color compensation and illumination compensation of at least one color component of a prediction for an image block of the picture without using explicit signaling. The method comprises at least the step of enabling the encoding of a picture described above in the apparatus, again without use of explicit signaling. In the description of an specific embodiment, this enabling of the color compensation and the illumination compensation is based on at least one of a group of pixels and a local signal block corresponding to the picture. Similar descriptions are provided for a method and apparatus for decoding the signal the encoded signal.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application claims the benefit, under 35 U.S.C. §365 ofInternational Application PCT/US2007/021803 filed Oct. 11, 2007, whichwas published in accordance with PCT Article 21(2) on Apr. 24, 2008 inEnglish and which claims the benefit of U.S. provisional patentapplication No. 60/852,530 filed Oct. 18, 2006. Furthermore, thisapplication is closely related in disclosed subject matter to aconcurrently filed National Stage application (U.S. patent applicationSer. No. 12/311,918 filed Apr. 17, 2009) having the same title whichapplication claims the benefit, under 35 U.S.C. §365 of InternationalApplication PCT/US2007/021812 filed Oct. 11, 2007, which was publishedin accordance with PCT Article 21(2) on Apr. 24, 2008 in English andwhich also claims the benefit of U.S. provisional patent application No.60/852,530 filed Oct. 18, 2006.

TECHNICAL FIELD

The present principles relate generally to video encoding and decodingand, more particularly, to methods and apparatus for local illuminationand color compensation without explicit signaling.

BACKGROUND

It has been widely recognized that Multi-view Video Coding (MVC) is atechnology that serves a wide variety of applications including, forexample, free-viewpoint and 3D video applications, home entertainmentand surveillance. In those multi-view applications, the amount of videodata involved is typically very large. Thus, there exists the need forefficient compression technologies to improve the coding efficiency ofcurrent video coding solutions performing simulcast of independentviews.

Since a multi-view video source includes multiple views of the samescene, there exists a high degree of correlation between the multipleview images. Therefore, view redundancy can be exploited in addition totemporal redundancy and is achieved by performing view prediction acrossthe different views.

In a practical scenario, multi-view video systems involving a largenumber of cameras will be built using heterogeneous cameras, or camerasthat have not been perfectly calibrated. This leads to differences inluminance and chrominance when the same parts of a scene are viewed withdifferent cameras. Moreover, camera distance and positioning alsoaffects illumination, in the sense that the same surface may reflect thelight differently when perceived from different angles. Under thesescenarios, luminance and chrominance differences will decrease theefficiency of cross-view prediction.

Several approaches have been proposed for solving the illuminationmismatch problem between pairs of images. In one such approach,hereinafter referred to as the first prior art approach, a scale/offsetparameter for a 16×16 macroblock and predictive coding of theseparameters may be used. Also, in the first prior approach, arate-distortion cost based enabling switch may be used. However, thefirst prior art approach focuses on temporal video sequences. In videosequences, the illumination mismatch problem does not typically occurconsistently as in cross-view prediction. In other prior art approaches,local illumination compensation methods for multi-view video coding areproposed, such as, for example, an approach in which an offset for eachsignal block is predictive coded and signaled in order to compensate theillumination differences in cross-view prediction.

May prior art approaches to illumination compensation use signaling bitsto achieve illumination compensation. The signaled information will beable to better represent illumination mismatches, but the extra overheadin sending that information will penalize the benefit of gaining betterprediction.

SUMMARY

These and other drawbacks and disadvantages of the prior art areaddressed by the present principles, which are directed to methods andapparatus for local illumination and color compensation without explicitsignaling.

According to an aspect of the present principles, there is provided anapparatus. The apparatus includes an encoder for encoding a picture byenabling at least one of color compensation and illuminationcompensation of at least one color component of a prediction for animage block of the picture without using explicit signaling.

According to another aspect of the present principles, there is provideda method. The method includes encoding a picture by enabling at leastone of color compensation and illumination compensation of at least onecolor component of a prediction for an image block of the picturewithout using explicit signaling.

According to still another aspect of the present principles, there isprovided an apparatus. The apparatus includes a decoder for decoding apicture by enabling at least one of color compensation and illuminationcompensation of at least one color component of a prediction for animage block of the picture without receipt of any explicit signalingcorresponding thereto.

According to a still further aspect of the present principles, there isprovided a method. The method includes encoding a picture by enabling atleast one of color compensation and illumination compensation of atleast one color component of a prediction for an image block of thepicture without receipt of any explicit signaling corresponding thereto.

According to a yet further aspect of the present principles, there isprovided an apparatus. The apparatus includes an encoder for encoding apicture by deriving at least one of illumination compensationinformation and color compensation information for a region in thepicture based on illumination information and color information from oneor more other regions in at least one of the picture and one or morerelated pictures.

According to an additional aspect of the present principles, there isprovided a method. The method includes encoding a picture by deriving atleast one of illumination compensation information and colorcompensation information for a region in the picture based onillumination information and color information from one or more otherregions in at least one of the picture and one or more related pictures.

According to a further additional aspect of the present principles,there is provided an apparatus. The apparatus includes a decoder fordecoding a picture by deriving at least one of illumination compensationinformation and color compensation information for a region in thepicture based on illumination information and color information from oneor more other regions in at least one of the picture and one or morerelated pictures.

According to a yet further additional aspect of the present principles,there is provided a method. The method includes decoding a picture byderiving at least one of illumination compensation information and colorcompensation information for a region in the picture based onillumination information and color information from one or more otherregions in at least one of the picture and one or more related pictures.

According to a still further aspect of the present principles, there isprovided an apparatus. The apparatus includes an encoder for encoding apicture by selectively determining whether to implicitly signal anenablement of at least one of illumination compensation and colorcompensation for at least one color component of a prediction for animage block of the picture.

According to a yet further aspect of the present principles, there isprovided a method. The method includes encoding a picture by selectivelydetermining whether to implicitly signal an enablement of at least oneof illumination compensation and color compensation for at least onecolor component of a prediction for an image block of the picture.

These and other aspects, features and advantages of the presentprinciples will become apparent from the following detailed descriptionof exemplary embodiments, which is to be read in connection with theaccompanying drawings.

BRIEF DESCRIPTION OF THE DRAWINGS

The present principles may be better understood in accordance with thefollowing exemplary figures, in which:

FIG. 1 is a block diagram for an exemplary Multi-view Video Coding (MVC)encoder to which the present principles may be applied, in accordancewith an embodiment of the present principles;

FIG. 2 is a block diagram for an exemplary Multi-view Video Coding (MVC)decoder to which the present principles may be applied, in accordancewith an embodiment of the present principles;

FIG. 3 is a flow diagram for an exemplary method for encoding a pictureusing at least one of illumination compensation and color compensation,in accordance with an embodiment of the present principles; and

FIG. 4 is a flow diagram for an exemplary method for decoding a pictureusing at least one of illumination compensation and color compensation,in accordance with an embodiment of the present principles.

DETAILED DESCRIPTION

The present principles are directed to methods and apparatus for localillumination and color compensation without explicit signaling.

The present description illustrates the present principles. It will thusbe appreciated that those skilled in the art will be able to devisevarious arrangements that, although not explicitly described or shownherein, embody the present principles and are included within its spiritand scope.

All examples and conditional language recited herein are intended forpedagogical purposes to aid the reader in understanding the presentprinciples and the concepts contributed by the inventor(s) to furtheringthe art, and are to be construed as being without limitation to suchspecifically recited examples and conditions.

Moreover, all statements herein reciting principles, aspects, andembodiments of the present principles, as well as specific examplesthereof, are intended to encompass both structural and functionalequivalents thereof. Additionally, it is intended that such equivalentsinclude both currently known equivalents as well as equivalentsdeveloped in the future, i.e., any elements developed that perform thesame function, regardless of structure.

Thus, for example, it will be appreciated by those skilled in the artthat the block diagrams presented herein represent conceptual views ofillustrative circuitry embodying the present principles. Similarly, itwill be appreciated that any flow charts, flow diagrams, statetransition diagrams, pseudocode, and the like represent variousprocesses which may be substantially represented in computer readablemedia and so executed by a computer or processor, whether or not suchcomputer or processor is explicitly shown.

The functions of the various elements shown in the figures may beprovided through the use of dedicated hardware as well as hardwarecapable of executing software in association with appropriate software.When provided by a processor, the functions may be provided by a singlededicated processor, by a single shared processor, or by a plurality ofindividual processors, some of which may be shared. Moreover, explicituse of the term “processor” or “controller” should not be construed torefer exclusively to hardware capable of executing software, and mayimplicitly include, without limitation, digital signal processor (“DSP”)hardware, read-only memory (“ROM”) for storing software, random accessmemory (“RAM”), and non-volatile storage.

Other hardware, conventional and/or custom, may also be included.Similarly, any switches shown in the figures are conceptual only. Theirfunction may be carried out through the operation of program logic,through dedicated logic, through the interaction of program control anddedicated logic, or even manually, the particular technique beingselectable by the implementer as more specifically understood from thecontext.

In the claims hereof, any element expressed as a means for performing aspecified function is intended to encompass any way of performing thatfunction including, for example, a) a combination of circuit elementsthat performs that function or b) software in any form, including,therefore, firmware, microcode or the like, combined with appropriatecircuitry for executing that software to perform the function. Thepresent principles as defined by such claims reside in the fact that thefunctionalities provided by the various recited means are combined andbrought together in the manner which the claims call for. It is thusregarded that any means that can provide those functionalities areequivalent to those shown herein.

Reference in the specification to “one embodiment” or “an embodiment” ofthe present principles means that a particular feature, structure,characteristic, and so forth described in connection with the embodimentis included in at least one embodiment of the present principles. Thus,the appearances of the phrase “in one embodiment” or “in an embodiment”appearing in various places throughout the specification are notnecessarily all referring to the same embodiment.

The phrase “multi-view video sequence” refers to a set of two or morevideo sequences that capture the same scene from different view points.

The phrases “cross-view” and “inter-view” are interchangeably usedherein to both refer to pictures that belong to a view other than acurrent view.

It is to be appreciated that the use of the term “and/or”, for example,in the case of “A and/or B”, is intended to encompass the selection ofthe first listed option (A), the selection of the second listed option(B), or the selection of both options (A and B). As a further example,in the case of “A, B, and/or C”, such phrasing is intended to encompassthe selection of the first listed option (A), the selection of thesecond listed option (B), the selection of the third listed option (C),the selection of the first and the second listed options (A and B), theselection of the first and third listed options (A and C), the selectionof the second and third listed options (B and C), or the selection ofall three options (A and B and C). This may be extended, as readilyapparent by one of ordinary skill in this and related arts, for as manyitems listed.

Moreover, it is to be appreciated that while one or more embodiments ofthe present principles are described herein with respect to the MPEG-4AVC standard, the present principles are not limited to solely thisstandard and, thus, may be utilized with respect to other video codingstandards, recommendations, and extensions thereof, including extensionssuch as multi-view (and non-multi-view) extensions of the MPEG-4 AVCstandard, while maintaining the spirit of the present principles.

Also, it is to be appreciated that while one or more embodiments of thepresent principles are described herein with respect to be applied to aMulti-view Video Coding (MVC) extension of the MPEG-4 AVC Standard, forwhich motion compensation and disparity compensation are enabled, thepresent principles are not limited to solely the preceding and, thus,may be utilized with respect to any Multi-view video Coding scheme forwhich disparity compensation is enabled.

Additionally, it is to be appreciated that the present principles may beadvantageously applied to video processing in addition to and/or inplace of video compression and/or video decompression. For example, thepresent principles may be readily used with any video processor and/orvideo processing that involves illumination and/or color compensation.

Further, it is to be appreciated that the present principles may beadvantageously applied to image registration and camera calibration.These and other applications of the present principles are readilydetermined by one of ordinary skill in this and related arts, given theteachings of the present principles provided herein.

Moreover, it is to be appreciated that the present principles may beapplied with respect to illumination and/or color compensation.

Further, it is to be appreciated that the present principles may be usedwith respect to multi-view and single-view video coding and decoding(e.g., temporal prediction in single-view video coding).

As used herein, the phrase “related pictures” refers to temporalreference pictures and/or cross-view reference pictures.

Moreover, as used herein, the phrase “local signal block” refers to aspatial neighboring block, and/or a view/temporal co-located/neighboringblock.

Also, while the proposed illumination and/or color compensationapproaches are described primarily with respect to an offset for eachsignal block, such approaches may be readily extended to include a scalefactor and/or a combination of an offset and scaling, as is readilyapparent to one of ordinary skill in this and related arts, whilemaintaining the spirit of the present principles.

Turning to FIG. 1, an exemplary Multi-view Video Coding (MVC) encoder towhich the present principles may be applied is indicated generally bythe reference numeral 100. The encoder 100 includes a combiner 105having an output connected in signal communication with an input of atransformer 110. An output of the transformer 110 is connected in signalcommunication with an input of quantizer 115. An output of the quantizer115 is connected in signal communication with an input of an entropycoder 120 and an input of an inverse quantizer 125. An output of theinverse quantizer 125 is connected in signal communication with an inputof an inverse transformer 130. An output of the inverse transformer 130is connected in signal communication with a first non-inverting input ofa combiner 135. An output of the combiner 135 is connected in signalcommunication with an input of an intra predictor 145 and an input of adeblocking filter 150. An output of the deblocking filter 150 isconnected in signal communication with an input of a reference picturestore 155 (for view i). An output of the reference picture store 155 isconnected in signal communication with a first input of a motioncompensator 175 and a first input of a motion estimator 180. An outputof the motion estimator 180 is connected in signal communication with asecond input of the motion compensator 175

An output of a reference picture store 160 (for other views) isconnected in signal communication with a first input of adisparity/illumination estimator 170 and a first input of adisparity/illumination compensator 165. An output of thedisparity/illumination estimator 170 is connected in signalcommunication with a second input of the disparity/illuminationcompensator 165.

An output of the entropy encoder 120 is available as an output of theencoder 100. A non-inverting input of the combiner 105 is available asan input of the encoder 100, and is connected in signal communicationwith a second input of the disparity/illumination estimator 170, and asecond input of the motion estimator 180. An output of a switch 185 isconnected in signal communication with a second non-inverting input ofthe combiner 135 and with an inverting input of the combiner 105. Theswitch 185 includes a first input connected in signal communication withan output of the motion compensator 175, a second input connected insignal communication with an output of the disparity/illuminationcompensator 165, and a third input connected in signal communicationwith an output of the intra predictor 145.

A mode decision module 140 has an output connected to the switch 185 forcontrolling which input is selected by the switch 185.

Turning to FIG. 2, an exemplary Multi-view Video Coding (MVC) decoder towhich the present principles may be applied is indicated generally bythe reference numeral 200. The decoder 200 includes an entropy decoder205 having an output connected in signal communication with an Input ofan inverse quantizer 210. An output of the inverse quantizer isconnected in signal communication with an input of an inversetransformer 215. An output of the inverse transformer 215 is connectedin signal communication with a first non-inverting input of a combiner220. An output of the combiner 220 is connected in signal communicationwith an input of a deblocking filter 225 and an input of an intrapredictor 230. An output of the deblocking filter 225 is connected insignal communication with an input of a reference picture store 240 (forview i). An output of the reference picture store 240 is connected insignal communication with a first input of a motion compensator 235.

An output of a reference picture store 245 (for other views) isconnected in signal communication with a first input of adisparity/illumination compensator 250.

An input of the entropy decoder 205 is available as an input to thedecoder 200, for receiving a residue bitstream. Moreover, an input of amode module 260 is also available as an input to the decoder 200, forreceiving control syntax to control which input is selected by theswitch 255. Further, a second input of the motion compensator 235 isavailable as an input of the decoder 200, for receiving motion vectors.Also, a second input of the disparity/illumination compensator 250 isavailable as an input to the decoder 200, for receiving disparityvectors and illumination compensation syntax.

An output of a switch 255 is connected in signal communication with asecond non-inverting input of the combiner 220. A first input of theswitch 255 is connected in signal communication with an output of thedisparity/illumination compensator 250. A second input of the switch 255is connected in signal communication with an output of the motioncompensator 235. A third input of the switch 255 is connected in signalcommunication with an output of the intra predictor 230. An output ofthe mode module 260 is connected in signal communication with the switch255 for controlling which input is selected by the switch 255. An outputof the deblocking filter 225 is available as an output of the decoder200.

As noted above, the present principles are directed to a method andapparatus for local illumination and color compensation without explicitsignaling. The present principles may be used with respect to multi-viewand single-view video sequences. Moreover, the present principles may beimplemented

In an embodiment, “implicit” illumination and/or color compensation canbe carried out for the local signals based on causally availableneighboring information without explicitly signaling any extrainformation.

A problem addressed by at least one embodiment is the efficient codingof multi-view video sequences. As noted above, a multi-view videosequence is a set of two or more video sequences that capture the samescene from different view points.

In an illustrative embodiment of the present principles involvingillumination compensation and/or color compensation, directed to theMulti-view Video Coding (MVC) extension of the MPEG-4 AVC standard, anexemplary framework is set forth as follows. At the slice level, newsyntax elements (ic_prediction_flag and/or cc_prediction_flag) areintroduced to indicate whether illumination compensation and colorcompensation are respectively enabled for the current slice. At themacroblock-level, new syntax elements are introduced: ic_enable and/orcc_enable are introduced to indicate the utilization of illuminationcompensation and color compensation, respectively, for each block; andic_sym and/or cc_sym are introduced to convey the illumination offsetparameter and color offset parameter, respectively. In summary,ic_enable and/or cc_enable, ic_sym, and/or cc_sym are the new syntaxelements that we need to signal for illumination and color compensation.

Thus, in an embodiment, we propose methods to enable illuminationcompensation and/or color compensation without the need to signal anyillumination and/or color compensation specific information in theprediction process of the local signal, i.e., we do not need to sendsyntax elements for ic_enable and/or cc_enable, ic_sym, and/or cc_sym.Those elements are derived using our invention.

In an embodiment, the illumination and/or color compensation is fullyintegrated with the variable block size functionalities in the MPEG-4AVC Standard.

Illumination Compensation as Part of the Cross-View Prediction Process

In an embodiment involving a coding application, illuminationcompensation will be considered as part of the disparity compensationprocess. In this scenario, cross-view prediction typically involves thecomputation of a disparity field between pictures from different views.The disparity field in cross-view prediction may be considered to beanalogous to what the motion field is in temporal prediction. For thesake of simplicity, we will presume in the following that cross-viewprediction, and by extension disparity estimation, is performed on ablock basis. The extension of the disclosed methods, features, andaspects to other groups of samples will be straight forward to those ofordinary skill in this and related arts. Such other groups of samplesinclude, for example, a single pixel, a block having more than onepixel, a frame, a picture, and so forth.

When illumination compensation is used in the disparity compensation ofa block, the illumination compensated reference block is calculated asfollows:B _(r)(x,y)=R(x+Δx,y+Δy)+ic_offsetwhere R(x,y) is the cross-view reference picture, and (Δx, Δy) is thedisparity vector (DV). ic_offset is the amount of offset applied to thereference block in order to address the mismatch between the currentsignal and the reference signal.Local Illumination Compensation Without Explicit Signaling

Presuming that one or more embodiments of the present principles areapplied to a Multi-view Video Coding (MVC) extension of the MPEG-4 AVCStandard, we can perform illumination compensation (IC) and/or colorcompensation (CC) without explicitly signaling any illuminationcompensation specific or color compensation specific information. Thiscan be achieved, for example, by utilizing the signal statistics in atemporal, view, and/or spatial neighborhood around the current signal,which is causally available (for example, which has been decoded) at thedecoder side. Usually signal statistics are highly correlated in atemporal, view, and/or spatial neighborhood, and this characteristic canbe utilized to derive the illumination compensation and/or colorcompensation information for the current signal without explicitsignaling. In particular, we use illumination compensation offsets as anexample of signal statistics in the following illustrations, and theillumination compensation information includes whether to enable IC andthe amount of IC to apply. Of course, the same applies to colorcompensation, which is not described partly herein for the sake ofbrevity.

In an embodiment, one possibility for using illumination compensation asan example of signal statistics is to use the previouslyprocessed/decoded illumination compensation offsets in the spatiallyneighboring blocks. The spatially neighboring blocks may also be fromother views, or from temporally-different pictures (same view or adifferent view). Whether to enable illumination compensation and theamount of illumination compensation used for the current block can bederived from the neighboring illumination compensation offsets byaveraging, median filtering, and/or other types of filtering. Note thatthe filtering may be, for example, linear or non-linear.

In another embodiment, another possibility for using illuminationcompensation as an example of signal statistics is to use theprocessed/decoded illumination compensation offsets in the co-locatedsignal blocks in previously processed/decoded pictures (temporallydifferent pictures). Whether to enable illumination compensation and theamount of illumination compensation used for the current block can thenbe derived from those co-located illumination compensation offsets byaveraging, median filtering, and/or other types of filtering. Theco-located signal block (as with the spatially neighboring blocks fromabove) can be from a previously processed/decoded picture in the sameview; or from a previously processed/decoded picture in other views.Also, conceivably the signal position from where the illuminationcompensation offsets are derived can be specified by a displacementvector. The displacement vector can either be signaled or derived fromspatial neighbors, and with this displacement vector a better correlatedsignal may be located in the picture to where it points.

Blocks (co-located, spatially neighboring, and/or otherwise) may be usedfrom current pictures in other view and even from future (temporallyadvanced) pictures. However, such implementations may introduceadditional processing and/or delays.

Since there is no explicit signaling for illumination compensationsyntax, the proposed implicit illumination compensation method can becombined with existing Skip and/or Direct modes in the MPEG-4 AVCStandard or any extension thereof. For Skip and Direct modes, theillumination compensation offset as well as motion information can bederived and then used to reconstruct the current signal. The ability toderive illumination compensation information without signaling overheadis very efficient in terms of compression efficiency. In addition tothis implicit illumination compensation mode, the derived illuminationcompensation information from temporal, view, and/or spatial neighborscan be used as a predictor in predictive coding of illuminationcompensation parameters in explicit illumination compensation mode.

In an embodiment directed to the MPEG-4 AVC Standard, the derivationprocesses of illumination compensation for macroblock-based Skip modeand Direct modes are illustrated as follows:

For P skip mode, whether illumination compensation is enabled (ic_flag)for the current macroblock and the amount of illumination compensationapplied (ic_offset) are derived from neighboring macroblocks. The syntaxic_flag is set to 1 if illumination compensation is enabled for eitherthe upper macroblock or the left macroblock, and conversely ic_flag isset to 0 if illumination compensation is enabled for neither the uppermacroblock nor the left macroblock. The average value of ic_offset fromthe upper and left macroblocks is used as the illumination compensationparameter of the current macroblock if both neighboring macroblocks useillumination compensation. If only one neighboring macroblock usesillumination compensation, the ic_offset of that macroblock is used forthe current macroblock. Mathematically, the illumination compensationprocess can be expressed as:B _(r)(x,y)=R(x+Δx,y+Δy)+ic_flag?ic_offset:0where both ic_flag and ic_offset are derived from neighboringmacroblocks without explicit signaling.

The same principle can essentially be applied to Direct modes in B_SLICEalso. For both Direct_16×16 and Direct_8×8 modes in B_SLICE, ic_flag andic_offset are derived from neighboring decoded signal blocks.

It is to be appreciated that other embodiments can be carried out in thetransform domain. More specifically, in an embodiment, the illuminationcompensation offset in the pixel domain is equivalent to the DiscreteCosine (DC) coefficient in the transform domain. The proposed implicitillumination compensation methods can then be carried out as follows:

(Step 1) The amount of IC of the neighboring blocks is extracted fromtheir respective DC coefficients in the transform domain;

(Step 2) Whether to enable illumination compensation and the amount ofillumination compensation used for the current block can then be derivedfrom the information obtained in step 1;

(Step 3) The illumination compensation process for the current block canbe carried out in the transform domain using the result of step 2.

It is to be appreciated that the transform domain based approach can becombined with the pixel-domain based approach. For example, in aparticular design, step (3) can be carried out in the pixel domain, butstep (1) can be performed in the transform domain.

It is to be appreciated that for Skip and Direct modes with derivedillumination compensation information, the motion vectors are derivedand, thus, there is no illumination compensation adaptive motionestimation involved at the encoder side. For these modes, the derivedillumination compensation information is applied to the reference blockpointed to by the derived motion vector.

In an MPEG-4 AVC Standard based embodiment, the existence of multiplereference pictures can be taken into account in the proposed implicitillumination compensation methods. For example, in order to utilize themost correlated illumination compensation information, the derivation ofillumination compensation offsets can be restricted, for example, toneighboring blocks that are predicted from the same reference picture,or to blocks that are predicted from one of multiple reference picturesthat are considered to be similar according to some metric.Alternatively, all blocks can be utilized (in some neighborhood, forexample) regardless of which reference picture from which the block ispredicted. Moreover, we can optionally apply the same rules thatdescribe how multiple reference pictures are considered for motionvector prediction.

The MPEG-4 AVC Standard supports variable block size motioncompensation, with block sizes ranging from 16×16 to 4×4. The proposedimplicit illumination compensation methods can be applied to one or moreblock sizes, depending on the nature of the illumination changes in thesignal and the encoder/decoder complexity allowed.

Also, the degree of illumination mismatch varies from one image toanother. Thus, it might not be efficient to always enable the proposedimplicit illumination compensation methods. To serve that purpose,sequence/picture/slice level syntaxes can be signaled to indicatewhether the proposed implicit illumination compensation method isutilized for the sequence/picture/slice.

An Encoder Design for the Proposed Implicit IC Methods

A description of an embodiment of an encoding method is provided tobetter illustrate how the proposed implicit illumination compensationmethods can be used. Of course, the present principles are not limitedto this encoding method and, given the teachings of the presentprinciples provided herein, one of ordinary skill in this and relatedarts will contemplate these and other encoding methods and variationsthereof, while maintaining the spirit of the present principles.Presuming we are currently coding block_i, and there exists severalneighboring blocks that enable illumination compensation, each with anIC offset of ic_offset_j. The offsets may be calculated, for example, asthe average, over all pixels in a block, of the difference between thepixels in the block and the corresponding pixels in the correspondingblock of the reference picture. The encoder has the flexibility tochoose whether or not to use the derived illumination compensationoffset, that is, to choose from the following two modes of operation.The choice may be conveyed to the decoder using one or more bits, or maybe implicitly conveyed without using any bits. For example, zero bitsmay be used by the decoder knowing, for example, when the received datacorresponds to a new block, and being able to implicitly determine thatthe previous block had no signaling information for illuminationcompensation and/or color compensation.

In an embodiment, the ic_offset_i is derived by utilizing theneighboring ic_offset_j, for example averaging them. The derivedic_offset_i can then in turn be used to derive IC for blocks later inthe processing order.

In another embodiment, ic_offset_i is calculated and signaled bycomputing the following:ic_offset_i=mean(B(x,y)−R(x+Δx,y+Δy))where B denotes the current block and R denotes the reference picture.The computed ic_offset_i could be further processed by predictive codingand/or quantization. In the case where quantization is involved, thereconstructed value ic_offset_recon_i will be used to deriveillumination compensation for blocks later in the processing order;otherwise the ic_offset_i will be used.

The encoder can choose from the above two modes of operation based oncertain criteria, for example rate distortion cost. The implicitillumination compensation method (option 1) might not be the best modein terms of compensating illumination differences. However, since theimplicit illumination compensation method (option 1) does not cost anyextra overhead, it might be better than the explicit illuminationcompensation method (option 2) in terms of rate distortion trade-off.

Various implementations may be made as, for example, a process/method,an apparatus, a set of instructions, a medium for carrying instructions,and a signal. Further, the various implementations, features, andaspects described herein may be combined to form additionalimplementations, and features may be added, omitted, and modified.Additionally, the headings in this disclosure are in no way intended tobe limiting or, for example, to limit the features described in onesection to just that section.

Turning to FIG. 3, an exemplary method for encoding a picture using atleast one of illumination compensation and color compensation isindicated generally by the reference numeral 300.

The method 300 includes a start block 305 that passes control to a looplimit block 310. The loop limit block 310 performs a loop over eachmacroblock in a current picture, including initializing a variable mb(having a value from 0 to MacroBlocksInSlice-1), and passes control to adecision block 315. The decision block determines whether or notillumination compensation and/or color compensation are enabled for thisslice. If so, then control is passed to a function block 320. Otherwise,control is passed to a function block 355.

The function block 320 performs motion estimation and mode decision withexplicit illumination compensation and/or explicit color compensation,obtains the rate-distortion (RD) cost, and passes control to a decisionblock 325. The decision block 325 determines whether or not implicitillumination compensation and/or implicit color compensation is enabledfor this mode. If so, then control is passed to a function block 330.Otherwise, control is passed to a function block 355.

The function block 330 derives whether or not to use illuminationcompensation and/or color compensation from spatial/temporal/cross-viewneighboring blocks, and passes control to a function block 335. Thefunction block 335 applies the derived illumination compensationinformation and/or color compensation information in the motioncompensation, obtains the rate-distortion (RD) cost, and passes controlto a function block 340. The function block 340 performs a modedecision, and passes control to a function block 345. The function block345 performs syntax writing, and passes control to a loop limit block350. The loop limit block 350 ends the loop over each macroblock in thecurrent picture, and passes control to an end block 399.

The function block 355 performs motion estimation and mode decisionwithout illumination compensation and color compensation, obtains therate-distortion (RD) cost, and passes control to the function block 340.

Turning to FIG. 4, an exemplary method for decoding a picture using atleast one of illumination compensation and color compensation isindicated generally by the reference numeral 400.

The method 400 includes a start block 405 that passes control to a looplimit block 410. The loop limit block 410 performs a loop over eachmacroblock in the current picture, including initializing a variable mb(having a value from 0 to MacroBlocksInSlice-1), and passes control to afunction block 415. The function block 415 performs syntax reading, andpasses control to a decision block 420. The decision block 420determines whether or not illumination compensation and/or colorcompensation are enabled for this slice. If so, then control is passedto a decision block 425. Otherwise, control is passed to a functionblock 445.

The decision block 425 determines whether or not implicit illuminationcompensation and/or implicit color compensation are enabled for thecurrent mode. If so, then control is passed to a function block 430.Otherwise, control is passed to a decision block 450.

The function block 430 derives whether to use illumination compensationand/or color compensation from the amount of illumination compensationand color compensation, respectively, from spatial/temporal/cross-viewneighboring blocks, and passes control to the function block 435. Thefunction block 435 applies the derived illumination compensation and/orcolor compensation information in the motion compensation withillumination compensation and/or color compensation, and passes controlto a loop limit block 440. The loop limit block 440 ends the loop overeach macroblock in the current picture, and passes control to an endblock 499.

The function block 445 performs motion compensation without illuminationcompensation and color compensation, and passes control to the looplimit block 440.

The decision block 450 determines whether or not illuminationcompensation and/or color compensation is enabled. If so, then controlis passed to a function block 455. Otherwise, control is passed to thefunction block 445.

The function block 455 forms an illumination compensation predictor,ic_offset_p, and/or a color compensation predictor, cc_offset_p, andpasses control to the function block 460. The function block 460 inversequantizes ic_sym and/or cc_sym, and differentially decodes ic_offsetand/or cc_offset, and passes control to a function block 465. Thefunction block 465 performs motion compensation with illuminationcompensation and/or color compensation, and passes control to the looplimit block 440.

A description will now be given of some of the many attendantadvantages/features of the present invention, some of which have beenmentioned above. For example, one advantage/feature is an apparatushaving an encoder for encoding a picture by enabling at least one ofcolor compensation and illumination compensation of at least one colorcomponent of a prediction for an image block of the picture withoutusing explicit signaling.

Another advantage/feature is the apparatus having the encoder asdescribed above, wherein the picture corresponds to multi-view contentfor a same or similar scene, single view content, and a scalable videolayer for the same scene.

Yet another advantage/feature is the apparatus having the encoder asdescribed above, wherein the encoder enables the at least one of thecolor compensation and the illumination compensation based on at leastone of a group of pixels and a local signal block corresponding to thepicture.

Moreover, another advantage/feature is the apparatus having the encoderthat enables the at least one of the color compensation and theillumination compensation based on at least one of a group of pixels anda local signal block corresponding to the picture as described above,wherein at least one of color compensation information and illuminationcompensation information for the at least one color component isrepresented as at least one of an offset and a scale factor.

Further, another advantage/feature is the apparatus having the encoderwherein at least one of color compensation information and illuminationcompensation information for the at least one color component isrepresented as at least one of an offset and a scale factor as describedabove, wherein the encoder uses a derivation process to derive the atleast one of the color compensation information and the illuminationcompensation information, the derivation process using signal statisticsin at least one of a spatial neighborhood, a view neighborhood, and atemporal neighborhood corresponding to the image block.

Also, another advantage/feature is the apparatus having the encoder thatuses the derivation process as described above, wherein at least one ofan averaging filter and a median filter is used in the derivationprocess.

Additionally, another advantage/feature is the apparatus having theencoder that uses the derivation process as described above, wherein thederivation process is performed in the transform domain.

Moreover, another advantage/feature is the apparatus having the encoderthat uses the derivation process as described above, wherein thederivation process includes evaluating multiple reference pictures usingduring an encoding of at least one of the image block and neighboringimage blocks with respect to the image block.

Further, another advantage/feature is the apparatus having the encoderthat uses the derivation process that evaluates multiple referencepictures as described above, wherein the derivation process uses theneighboring image blocks predicted from a same reference picture fromamong the multiple reference pictures.

Also, another advantage/feature is the apparatus having the encoder thatuses the derivation process as described above, wherein the at least oneof the derived color compensation information and the derivedillumination compensation information comprises an indication of whetherto enable the at least one of the color compensation and theillumination compensation, and an amount of the at least one of thecolor compensation and the illumination compensation to be applied tothe at least one color component.

Additionally, another advantage/feature is the apparatus having theencoder that uses the derivation process as described above, wherein theat least one of the derived color compensation information and thederived illumination compensation information is applied to a mode ofthe International Organization for Standardization/InternationalElectrotechnical Commission (ISO/IEC) Moving Picture Experts Group-4(MPEG-4) Part 10 Advanced Video Coding (AVC) standard/InternationalTelecommunication Union, Telecommunication Sector (ITU-T) H.264recommendation.

Moreover, another advantage/feature is the apparatus having the encoderthat uses the derivation process as described above, wherein the atleast one of the derived color compensation information and the derivedillumination compensation information is applied to an existing Skipmode in the International Organization for Standardization/InternationalElectrotechnical Commission (ISO/IEC) Moving Picture Experts Group-4(MPEG-4) Part 10 Advanced Video Coding (AVC) standard/InternationalTelecommunication. Union, Telecommunication Sector (ITU-T) H.264recommendation.

Further, another advantage/feature is the apparatus having the encoderthat uses the derivation process as described above, wherein the atleast one of the derived color compensation information and the derivedillumination compensation information is applied to an existing Directmode in the International Organization for Standardization/InternationalElectrotechnical Commission (ISO/IEC) Moving Picture Experts Group-4(MPEG-4) Part 10 Advanced Video Coding (AVC) standard/InternationalTelecommunication Union, Telecommunication Sector (ITU-T) H.264recommendation.

Also, another advantage/feature is the apparatus having the encoder thatuses the derivation process as described above, wherein the at least oneof the derived color compensation information and the derivedillumination compensation information is applied to the at least onecolor component.

Additionally, another advantage/feature is the apparatus having theencoder as described above, wherein the at least one of the colorcompensation and the illumination compensation is enabled using at leastone of implicit signaling and explicit signaling.

Additionally, another advantage/feature is the apparatus having theencoder wherein the at least one of the color compensation and theillumination compensation is enabled using at least one of implicitsignaling and explicit signaling as described above, wherein the encoderrenders a decision to use at least one of the explicit signaling and theimplicit signaling based on a predetermined criterion.

Additionally, another advantage/feature is the apparatus having theencoder that renders a decision to use at least one of the explicitsignaling and the implicit signaling based on a predetermined criterionas described above, wherein the predetermined criterion comprises a ratedistortion cost.

Moreover, an advantage/feature is an apparatus having an encoder forencoding a picture by deriving at least one of illumination compensationinformation and color compensation information for a region in thepicture based on illumination information and color information from oneor more other regions in at least one of the picture and one or morerelated pictures.

Further, another advantage/feature is the apparatus having the encoderas described above, wherein the region in the picture is an image block,and the one or more other regions include at least one of spatiallyneighboring image blocks and temporally neighboring image blocks from asame view or a different view.

Also, an advantage/feature is an apparatus having an encoder forencoding a picture by selectively determining whether to implicitlysignal an enablement of at least one of illumination compensation andcolor compensation for at least one color component of a prediction foran image block of the picture.

Additionally, another advantage/feature is an apparatus having anencoder as described above, wherein the encoder derives at least one ofillumination compensation information and color compensation informationfor the at least one color component of the prediction for the imageblock in the picture based on illumination information and colorinformation for at least one of one or more other image blocks in thepicture and one or more other image blocks in related pictures,calculates at least one of illumination compensation information andcolor compensation information for the at least one color component,performs a comparison of at least one of the derived illuminationcompensation information to the calculated illumination information andthe derived color compensation information to the calculated colorcompensation information, and determines whether to signal theenablement of the at least one of the illumination compensation and thecolor compensation based on a result of the comparison.

These and other features and advantages of the present principles may bereadily ascertained by one of ordinary skill in the pertinent art basedon the teachings herein. It is to be understood that the teachings ofthe present principles may be implemented in various forms of hardware,software, firmware, special purpose processors, or combinations thereof.

Most preferably, the teachings of the present principles are implementedas a combination of hardware and software. Moreover, the software may beimplemented as an application program tangibly embodied on a programstorage unit. The application program may be uploaded to, and executedby, a machine comprising any suitable architecture. Preferably, themachine is implemented on a computer platform having hardware such asone or more central processing units (“CPU”), a random access memory(“RAM”), and input/output (“I/O”) interfaces. The computer platform mayalso include an operating system and microinstruction code. The variousprocesses and functions described herein may be either part of themicroinstruction code or part of the application program, or anycombination thereof, which may be executed by a CPU. In addition,various other peripheral units may be connected to the computer platformsuch as an additional data storage unit and a printing unit.

It is to be further understood that, because some of the constituentsystem components and methods depicted in the accompanying drawings arepreferably implemented in software, the actual connections between thesystem components or the process function blocks may differ dependingupon the manner in which the present principles are programmed. Giventhe teachings herein, one of ordinary skill in the pertinent art will beable to contemplate these and similar implementations or configurationsof the present principles.

Although the illustrative embodiments have been described herein withreference to the accompanying drawings, it is to be understood that thepresent principles is not limited to those precise embodiments, and thatvarious changes and modifications may be effected therein by one ofordinary skill in the pertinent art without departing from the scope orspirit of the present principles. All such changes and modifications areintended to be included within the scope of the present principles asset forth in the appended claims.

The invention claimed is:
 1. An apparatus, comprising: an encoder forencoding a picture by enabling at least one of color compensation andillumination compensation of at least one color component of aprediction for an image block of the picture without using explicitsignaling, the at least one of the color compensation and theillumination compensation being respectively performed to compensate forcolor mismatches and illumination mismatches between the image block andthe prediction for the image block.
 2. The apparatus of claim 1, whereinat least one of color compensation information and illuminationcompensation information for the at least one color component isrepresented as at least one of an offset and a scale factor, the offsetfor the image block being derived from a neighboring offset of aneighboring block, the offset and the neighboring offset being at leastone of a color compensation offset and an illumination compensationoffset that represent respective amounts of offset.
 3. The apparatus ofclaim 1, wherein said encoder enables the at least one of the colorcompensation and the illumination compensation based on at least one ofa group of pixels and a local signal block corresponding to the picture.4. The apparatus of claim 3, wherein at least one of color compensationinformation and illumination compensation information for the at leastone color component is represented as at least one of an offset and ascale factor, the offset being at least one of a color compensationoffset and an illumination compensation offset that represents an amountof offset.
 5. The apparatus of claim 4, wherein said encoder uses aderivation process to derive the at least one of the color compensationinformation and the illumination compensation information, thederivation process using signal statistics in at least one of a spatialneighborhood, a view neighborhood, and a temporal neighborhoodcorresponding to the image block.
 6. The apparatus of claim 5, whereinthe at least one of the derived color compensation information and thederived illumination compensation information comprises an indication ofwhether to enable the at least one of the color compensation and theillumination compensation, and an amount of the at least one of thecolor compensation and the illumination compensation to be applied tothe at least one color component.
 7. A method, comprising: encoding apicture by enabling at least one of color compensation and illuminationcompensation of at least one color component of a prediction for animage block of the picture without using explicit signaling, the atleast one of the color compensation and the illumination compensationbeing respectively performed to compensate for color mismatches andillumination mismatches between the image block and the prediction forthe image block.
 8. The method of claim 7, wherein at least one of colorcompensation information and illumination compensation information forthe at least one color component is represented as at least one of anoffset and a scale factor, the offset for the image block being derivedfrom a neighboring offset of a neighboring block, the offset and theneighboring offset being at least one of a color compensation offset andan illumination compensation offset that represent respective amounts ofoffset.
 9. The method of claim 7, wherein said enabling step enables theat least one of the color compensation and the illumination compensationbased on at least one of a group of pixels and a local signal blockcorresponding to the picture.
 10. The method of claim 9, wherein atleast one of color compensation information and illuminationcompensation information for the at least one color component isrepresented as at least one of an offset and a scale factor, the offsetbeing at least one of a color compensation offset and an illuminationcompensation offset that represents an amount of offset.
 11. The methodof claim 10, where said enabling step uses a derivation process toderive the at least one of the color compensation information and theillumination compensation information, the derivation process usingsignal statistics in at least one of a spatial neighborhood, a viewneighborhood, and a temporal neighborhood corresponding to the imageblock.
 12. The method of claim 11, wherein the at least one of thederived color compensation information and the derived illuminationcompensation information comprises an indication of whether to enablethe at least one of the color compensation and the illuminationcompensation, and an amount of the at least one of the colorcompensation and the illumination compensation to be applied to the atleast one color component.
 13. An apparatus, comprising: a decoder fordecoding a picture by enabling at least one of color compensation andillumination compensation of at least one color component of aprediction for an image block of the picture without receipt of anyexplicit signaling corresponding thereto, the at least one of the colorcompensation and the illumination compensation being respectivelyperformed to compensate for color mismatches and illumination mismatchesbetween the image block and the prediction for the image block.
 14. Theapparatus of claim 13, wherein at least one of color compensationinformation and illumination compensation information for the at leastone color component is represented as at least one of an offset and ascale factor, the offset for the image block being derived from aneighboring offset of a neighboring block, the offset and theneighboring offset being at least one of a color compensation offset andan illumination compensation offset that represent respective amounts ofoffset.
 15. The apparatus of claim 13, wherein said decoder enables theat least one of the color compensation and the illumination compensationbased on at least one of a group of pixels and a local signal blockcorresponding to the picture.
 16. The apparatus of claim 15, wherein atleast one of color compensation information and illuminationcompensation information for the at least one color component isrepresented as at least one of an offset and a scale factor, the offsetbeing at least one of a color compensation offset and an illuminationcompensation offset that represents an amount of offset.
 17. Theapparatus of claim 16, where said decoder uses a derivation process toderive the at least one of the color compensation information and theillumination compensation information, the derivation process usingsignal statistics in at least one of a spatial neighborhood, a viewneighborhood, and a temporal neighborhood corresponding to the imageblock.
 18. The apparatus of claim 17, wherein the at least one of thederived color compensation information and the derived illuminationcompensation information comprises an indication of whether to enablethe at least one of the color compensation and the illuminationcompensation, and an amount of the at least one of the colorcompensation and the illumination compensation to be applied to the atleast one color component.
 19. A method, comprising: decoding a pictureby enabling at least one of color compensation and illuminationcompensation of at least one color component of a prediction for animage block of the picture without receipt of any explicit signalingcorresponding thereto, the at least one of the color compensation andthe illumination compensation being respectively performed to compensatefor color mismatches and illumination mismatches between the image blockand the prediction for the image block.
 20. The method of claim 19,wherein at least one of color compensation information and illuminationcompensation information for the at least one color component isrepresented as at least one of an offset and a scale factor, the offsetfor the image block being derived from a neighboring offset of aneighboring block, the offset and the neighboring offset being at leastone of a color compensation offset and an illumination compensationoffset that represent respective amounts of offset.
 21. The method ofclaim 19, wherein said enabling step enables the at least one of thecolor compensation and the illumination compensation based on at leastone of a group of pixels and a local signal block corresponding to thepicture.
 22. The method of claim 21, wherein at least one of colorcompensation information and illumination compensation information forthe at least one color component is represented as at least one of anoffset and a scale factor, the offset being at least one of a colorcompensation offset and an illumination compensation offset thatrepresents an amount of offset.
 23. The method of claim 22, where saidenabling step uses a derivation process to derive the at least one ofthe color compensation information and the illumination compensationinformation, the derivation process using signal statistics in at leastone of a spatial neighborhood, a view neighborhood, and a temporalneighborhood corresponding to the image block.
 24. The method of claim23, wherein the at least one of the derived color compensationinformation and the derived illumination compensation informationcomprises an indication of whether to enable the at least one of thecolor compensation and the illumination compensation, and an amount ofthe at least one of the color compensation and the illuminationcompensation to be applied to the at least one color component.
 25. Anon-transitory storage media having video signal data encoded thereupon,comprising: a picture encoded by enabling at least one of colorcompensation and illumination compensation of at least one colorcomponent of a prediction for an image block of the picture without theuse of explicit signaling, the at least one of the color compensationand the illumination compensation being respectively performed tocompensate for color mismatches and illumination mismatches between theimage block and the prediction for the image block.
 26. Thenon-transitory storage media of claim 25, wherein at least one of colorcompensation information and illumination compensation information forthe at least one color component is represented as at least one of anoffset and a scale factor, the offset for the image block being derivedfrom a neighboring offset of a neighboring block, the offset and theneighboring offset being at least one of a color compensation offset andan illumination compensation offset that represent respective amounts ofoffset.
 27. The storage media of claim 25, wherein said video signaldata enabling the at least one of the color compensation and theillumination compensation is based on at least one of a group of pixelsand a local signal block corresponding to the picture.
 28. The storagemedia of claim 27, wherein at least one of color compensationinformation and illumination compensation information for the at leastone color component is represented as at least one of an offset and ascale factor, the offset being least one of a color compensation offsetand an illumination compensation offset that represents an amount ofoffset.
 29. The storage media of claim 28, wherein the at least one ofthe color compensation information and the illumination compensationinformation is derived; the derivation process using signal statisticsin at least one of a spatial neighborhood, a view neighborhood, and atemporal neighborhood corresponding to the image block.
 30. The storagemedia of claim 29, wherein the at least one of the derived colorcompensation information and the derived illumination compensationinformation comprises an indication of whether to enable the at leastone of the color compensation and the illumination compensation, and anamount of the at least one of the color compensation and theillumination compensation to be applied to the at least one colorcomponent.